Exercise Type 5: Model Comparison & Bayes Factor
What the exam asks: You have two competing models. After observing data, which model is more plausible? Compute the Bayes Factor and/or the ratio of posterior model probabilities.
Part 0: What Do All These Symbols Mean?
The Key Notation
| Symbol | How to Read It | What It Means |
|---|---|---|
| $p(D|m_1)$ | "probability of data given model 1" | The evidence for model 1 — how well model 1 predicted the data |
| $B_{12}$ | "Bayes Factor one-two" | Ratio of evidences: $B_{12} = p(D|m_1) / p(D|m_2)$ |
| $p(m_1|D)$ | "probability of model 1 given data" | How plausible model 1 is AFTER seeing the data |
| $p(m_1)$ | "probability of model 1" | How plausible model 1 was BEFORE seeing the data (prior) |
| $\frac{p(m_1|D)}{p(m_2|D)}$ | "ratio of posterior probabilities" | After seeing data, how many times more likely is model 1 vs model 2? |
The Key Concept: Bayes Factor
The Bayes Factor tells you which model explained the data better:
In plain English: "How many times better did model 1 explain the data compared to model 2?"
- If $B_{12} = 3$: Model 1 explained the data 3 times better than model 2
- If $B_{12} = 0.5$: Model 2 explained the data 2 times better than model 1
- If $B_{12} = 1$: Both models explained the data equally well
The Key Concept: Posterior Model Probability
After seeing data, how plausible is each model?
In plain English: "Model plausibility = (how well it explained the data) × (how plausible it was to begin with), divided by a normalizing number."
The Key Concept: Ratio of Posterior Probabilities
This is the MOST useful formula:
In plain English: "The ratio of posterior probabilities = (ratio of evidences) × (ratio of prior probabilities)"
Why this is useful: You don't need to compute $p(D)$ (the overall evidence). It cancels out!
Part 1: Key Formulas (MEMORIZE)
Formula 1: Evidence for a Model
Formula 2: Bayes Factor
Formula 3: Posterior Probability Ratio
Formula 4: Bayes Factor in Terms of Posteriors
This is just a rearrangement of Formula 3.
Part 2: FULL Walkthrough of Real Exam Questions
EXAM QUESTION 1 (2021-Part-B, Questions 2d-2e)
Model $m_1$: $p(x|\theta, m_1) = (1-\theta)\theta^x$, $p(\theta|m_1) = 6\theta(1-\theta)$ Model $m_2$: $p(x|\theta, m_2) = (1-\theta)\theta^x$, $p(\theta|m_2) = 2\theta$ Model priors: $p(m_1) = 2/3$, $p(m_2) = 1/3$
We observe $x = 4$.
Question 2d: Which model has the largest evidence?
STEP-BY-STEP SOLUTION
Step 1: Compute evidence for $m_1$
Likelihood at x=4: $(1-\theta)\theta^4$ Prior: $6\theta(1-\theta)$ Product: $6\theta^5(1-\theta)^2$
Using Beta function (p=5, q=2):
Step 2: Compute evidence for $m_2$
Likelihood at x=4: $(1-\theta)\theta^4$ Prior: $2\theta$ Product: $2\theta^5(1-\theta)$
Using Beta function (p=5, q=1):
Step 3: Compare
$p(x=4|m_1) = 1/28 \approx 0.0357$ $p(x=4|m_2) = 1/21 \approx 0.0476$
$1/21 > 1/28$, so model 2 has larger evidence.
Answer: (b) $m_2$ ✅
Question 2e: Which model has the largest posterior probability?
STEP-BY-STEP SOLUTION
Step 1: Write the posterior probability ratio formula
Step 2: Plug in the numbers
Step 3: Compute the evidence ratio
Step 4: Compute the prior ratio
Step 5: Multiply
Step 6: Interpret
Since the ratio is > 1, $p(m_1|x=4) > p(m_2|x=4)$. Model 1 has higher posterior probability.
Answer: (a) $m_1$ ✅
IMPORTANT INSIGHT
Model 2 had higher evidence, but model 1 has higher posterior! Why? Because model 1 had a much higher prior (2/3 vs 1/3), and this prior advantage outweighed the evidence disadvantage.
EXAM QUESTION 2 (2022, Question 1c — Bayes Factor Formula)
Which expression correctly gives the Bayes Factor $B_{12}$?
Options: - (a) $B_{12} = \frac{p(D|m_1)}{p(D|m_2)} = \frac{p(m_1|D)}{p(m_2|D)} \cdot \frac{p(m_1)}{p(m_2)}$ - (b) $B_{12} = \frac{p(D|m_1)}{p(D|m_2)} = \frac{p(m_1|D)}{p(m_2|D)} \cdot \frac{p(m_2)}{p(m_1)}$ - (c) $B_{12} = \frac{p(m_1|D)}{p(m_2|D)} = \frac{p(D|m_1)}{p(D|m_2)} \cdot \frac{p(m_2)}{p(m_1)}$ - (d) $B_{12} = \frac{p(m_1|D)}{p(m_2|D)} = \frac{p(D|m_1)}{p(D|m_2)} \cdot \frac{p(m_1)}{p(m_2)}$
STEP-BY-STEP SOLUTION
Step 1: The definition of Bayes Factor
This eliminates (c) and (d) because they define $B_{12}$ as the posterior ratio.
Step 2: The relationship between posterior ratio and Bayes Factor
Rearranging:
Step 3: Match the answer
(b) says $B_{12} = \frac{p(D|m_1)}{p(D|m_2)} = \frac{p(m_1|D)}{p(m_2|D)} \cdot \frac{p(m_2)}{p(m_1)}$ — correct!
Answer: (b) ✅
EXAM QUESTION 3 (2022, Question 4e)
$p(x=1|m_1) = 1/2$, $p(x=1|m_2) = 1/3$ $p(m_1) = 1/3$, $p(m_2) = 2/3$
Compute $\frac{p(m_1|x=1)}{p(m_2|x=1)}$.
Options: (a) $3/4$, (b) $4/9$, (c) $5/9$, (d) $2/3$
STEP-BY-STEP SOLUTION
Answer: (a) 3/4 ✅
Part 3: Tricks & Shortcuts
TRICK 1: Bayes Factor Is Just a Ratio of Evidences
Compute each evidence using the Beta function. Divide them.
TRICK 2: Posterior Ratio = Bayes Factor × Prior Ratio
Don't compute each posterior separately. Just multiply.
TRICK 3: Which Model Wins?
- If posterior ratio > 1: model 1 wins
- If posterior ratio < 1: model 2 wins
TRICK 4: The Bayes Factor Formula Pattern
$B_{12}$ = evidence ratio × (prior of model 2 / prior of model 1)
Watch for which prior goes on top — it's the OPPOSITE of what you might expect!
Part 4: Practice Exercises
Exercise 1
Model $m_1$: $p(x|\theta) = (1-\theta)\theta^x$, $p(\theta) = 6\theta(1-\theta)$ Model $m_2$: $p(x|\theta) = (1-\theta)\theta^x$, $p(\theta) = 2\theta$
After observing $x=4$, which model has larger evidence?
Options: - (a) $m_1$ - (b) $m_2$ - (c) Same evidence
Exercise 2
Same models. Priors: $p(m_1) = 2/3$, $p(m_2) = 1/3$.
After observing $x=4$, which model has larger posterior probability?
Options: - (a) $m_1$ - (b) $m_2$ - (c) Equal
Exercise 3
Model $m_1$: $p(x|\theta) = \theta^x(1-\theta)^{1-x}$, $p(\theta) = 6\theta(1-\theta)$ Model $m_2$: $p(x|\theta) = (1-\theta)^x\theta^{1-x}$, $p(\theta) = 1$ (uniform) $p(m_1) = 2/3$, $p(m_2) = 1/3$
Compute $\frac{p(m_1|x=1)}{p(m_2|x=1)}$.
Options: - (a) $1/3$ - (b) $1/2$ - (c) $2/3$ - (d) $2$
Exercise 4
$p(x=1|m_1) = 1/2$, $p(x=1|m_2) = 1/3$ $p(m_1) = 1/3$, $p(m_2) = 2/3$
Compute $\frac{p(m_1|x=1)}{p(m_2|x=1)}$.
Options: - (a) $3/4$ - (b) $4/9$ - (c) $5/9$ - (d) $2/3$